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ABSTRACT 

* ; 

i 
I 

A study vas conducted to deteifmine if the tendency for faculty members 
of higher rank to receive the highest ratings on the Illinois Course Evalua- 
tion Questionnaire (CEQ) remained when variables such as class size and 
course level were taken into account. The relationship between CEQ ratings 
and instructor's ranic, class siae, and level of course was examined by means 
of multivariate analysis of variance OMNOVA) • Dependent variables were the 
six subscales of the CEQ. As hypothesized there were no significant differ- 
ences in ratings assigned by students in small (1-20 students), mediixa (21*- 
40 students) , and large (over 40 students) classes (p < .13) » or received by 
teaching assistants, instructors, and assistant, associate, and full pro- 
fessors (p<a3). Highly significant differentes (p<.0001), however, were 
found in ratings assigned by students in 100- , 200- , 300- , and 400- and 500- 
level courses. In addition, significant size by level (p < .03) and size by 
rank (p < .02) interaction effects were found • Discriminant functions com- 
puted for effects found to be significant yielded information concerning the 
extent and direction of these significant differences. 



The Relationship betveen CEQ Ratings and Instructor's 
Rank, Class Size, and Course Level 
by 

Margaret H. Graham 

During recent years, there has been an increased awareness of the need 
for systematic investigation into the area of teaching effectiveness , parti- 
cularly at the college level. There are a number of ways in which such effect- 
tiveness can be assessed. Methods commonly enq>loyed at various institutions 
include classroom visitation, interviewing of students, ratings of faculty 
by colleagues, by students, by alumni, and by department chairmen, teacher 
self*evaluation, and even the appraisal of student achievement. Extensive 
discussion of methodological considerations involved in evaluating courses 
and/or instructors will not be included in this report since relevant 
consideration of such areas can be found in sources such as Anikeef (1953), 
Bayley (1967) , Bogue (1967) , Bryant (1967) , and Weaver (1960) , as well as 
more recent works and also journals devoted specifically to the evaluation 
of teacher effectiveness, such as Improving College and University Teaching 
published by the Oregon State University Press* 

One particular instrument currently being used to assess teaching 
effectiveness at the University of Illinois and elsewhere is the Illinois 
Course Evaluation Questionnaire (CEQ) . The CEQ was designed to gather 
student reactions to a "standardized set of statements relative to certain 
standardized aspects of an instructional program.*' It has norms which 
"enable an instructor to adequately compare his results with results of 
other instructors'* (Spencer imd.Ala^Aoill, 1969, p. 2). Through a proce<» 
dure involving the use of factor analysis and logical grouping of items, 
six subscales (General Course Attitude, Method of Instruction, Course 
Content, Interest-*Attention, Instructor, and Specific Items) were developed^ 
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The instrument consists of 50 items (Bach instructor can also include additional 
items of his choice.) to uhich students indicate their degree of agreement or 
disagreement* Responses to all CEQ items are based on a common four-point 
scale of strongly agree (SA) , agree (A) , disagree (D) , and strongly disagree 
(SD). The mean item response is calculated as the numerical average of the 
SAy A, D, and SD responses, with a laeight of 4 assigned to the most favorable 
response doim to a weight of 1 for the least favorable response. In some 
cases, SA is most favorable; in others, SD occupies that position. The 
higher the raw score on the CEQ, the more favorable is the attitude being 
measured. Since some CEQ items reflect positive attitudes and some reflect 
negative attitudes, weights for negatively-stated items are the inversion of 
those for positively-stated items. 

Administration of the CEQ by instructors is voluntary* CEQ results are 
confidential, and are presented only to the instructor who administers the 
instrument. Not only do such results give him insight into the manner in 
which he and his course are perceived by his students and thus suggest ways 
in which aspects of his program might be profitably altered, but they provide 
the Measurement and Research Division of the Office of Instructional Resources 
with information concerning instruction at the University as a whole, leading 
to constant updating of the various norms for the CEQ and the determination 
of characteristics such as validity and reliability of the instrument itself. 

Results on the CEQ have been studied in terms of variables such as level 
of course (Singhal, 1968; Mclnnis, 1966), content of the course (Slngbal, 1968), 
sex of the instructor (Singhal, 1968; Spencer, 1969; Singhal, 1968), "nl^ 
of the instructor (Stallings and Singhal, 1969; Singhal, 1968; Spencer, 1969), 
Instructor's severity of grading (Singhal and Stallings, 1967), class size 
(Spencer, 1969), grade expected by the student (Spencer, 1969; Singhal and 
Stallings, 1967), student status— freshman, sophomore, Junior, senior. 



graduate student, other-- (Spencer, 1969), and whether or not respondents have 
identified themselves (Spencer, 1965) . Higher instructor rank, higher course 
level, higher student status, higher expected grade, and identification of 
respondent have each tended to result in a more favorable student evaluation* 
Of particular interest to this investigator is the tendency for faculty 
members of higher rank to receive the highest ratings. To Mhat can these 
differences associated with rank be attributed? 

STATEMENT OF PROBLEM 
The purpose of this study is to attempt to determine if the tendency for 
faculty members of higher rank to receive the highest ratings on the CEQ 
remains when such variables as class size and course level are taken into 
account. Studies of the variable class size alone have not yielded signifi- 
cant results in terms of CEQ ratings. However, there might be an interaction 
between class size and instructor's ranlc. For example high rank instructors 
might be more favorably rated in larger classes, while low rank instructors 
might be more favorably rated in smaller classes. Those in smaller classes 
might value highly the greater opportunity for personal interaction with the 
instructor, thus making more positive evaluations, while those in larger 
classes might be favorably impressed at being taught by a person of high 
esteem, thus affecting their ratings. In terms of course level, perhaps 
high rank instructors receive higher ratings when teaching upper level courses 
and lower ratings when teaching lower level courses. Such a finding would 
be realistic since students in upper level courses, by virtue of the fact 
that they have remained at the University, tend to be of higher quality than 
those of lower level courses. Better students probably expect higher grades, 
and higher expected grades have been associated with higher ratings on the CEQ* 



METHOD 

In this study y the relationship between CEQ ratings and instructor's 
rank, class size, and level of course (all of uhich are probably correlated) 
will be examined by means of multivariate analysis of variance (M^NOVA) . 
Jones (1966) and Bock (1966) are among those persons who have vjorked exten- 
sively with this technique. Bock (1966, p. 821) pointed out that MANOVA 
and discriminant analysis are generalizations of methods of regression 
analysis, with MANOVA the generalization of the usual univariate ANOVA. 
MANOVA "formally includes the procedures of discriminant analysis but 
not its purpose" (Bock, 1966, p. 822). 

Jones (1966) indicated advantages and disadvantages of using MANOVA 
instead of alternative techniques (e.g., correlational). He stated that, 
whereas the correlational techniques are often used to estimate the magnitude 
of association between variables of interest, MANOVA, like univariate ANOVA, 
is used to determine the existence of significant differences between groups 
or conditions. MANOVA, however, is used in studying differences among 
groups on many variables simultaneously. For each significant main effect, 
discriminant functions are computed. [See Tatsuoka's (1970) booklet for an 
excellent and relatively simple discussion of discriminant analysis.] The 
functions obtained indicate the extent and direction of the differences 
found to be significant. Thus a valuable feature of MANOVA is the use of 
discriminant analysis in interpreting significant effects and suggesting 
hypotheses for further study in an area. 

A major problem encountered when using MANOVA concerns meeting assump- 
tions analagous to those of univariate ANOVA. Computations required in 
MANOVA, as with all statistical techniques, can be made without considering 
these assumptions. However, if valid inferences are to be made from results 



obtained, it is necessary to assume "that the uithin-cell residuals have a 
multivariate normal distribution with a common covariance matrix, and that 
observations on different individuals are uncorrelated" (Jones, 1966, p. 247). 
The latter assumption can be insured through experimental design. The former 
two assumptions (of normality of distribution and of homogeneity of covariance) 
are very difficult to prove in HANOVA, and unlike the case with univariate 

ANOVA, it cannot be assumed that violations of these assumptions are not 

t 

critical (Jones, 1966). 

Various criteria have been proposed for use in significance testing in 
MANOVA. Jones (1966) outlined advantages and disadvantages of the following 
three criteria and listed situations in vjhich each criterion is appropriate: 
Willcs' maximum likelihood criterion, Hotelling^s trace criterion, and Roy*s 
largest root criterion. Bartlett's chi-square criterion has also been widely 
used. 

Variables and Sampling 

The primary sampling unit in this study is course sections. The six 
dependent variables are section means of the sub-scales on the CEQ as 
indicated in Table 1. Suli-scale means for each course section v;ere computed 
so that the highest score possible was 4.00, with high scores indicating 
positive student attitudes toward the particular aspects of the course and 
instructor. The three independent variables are class size (factor A), 
course level (factor B) , and instructor ranlc (factor C) for each section. 



Class si2e > factor consists of three levels*: 
Level 1 « small (1-2C students) 
Level 2 » medium (21-40 students) 
Level 3 = large (over 40 students) • 

Course level , factor consists of four levels: 
Level 1 = 100-level courses 
Level 2 « 200 -level courses 
Level 3 = 300-level courses 
Level 4 » 400- and 500-level courses. 

Instructor rank , factor C , consists of five levels: 
Level 1 ^ teaching assistant 
Level 2 ^ instructor 
Level 3 « assistant professor 
Level 4 - associate professor 
Level 5 * full professor. 

Persons of rank other than the above (e.g., visiting lecturer^ visiting 
associate professor, etc.) are not included in this study. 



^lass size vas considered to be the number of students in each section 
responding to the questionnaire. 
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Tabic 1 

Definition of the Six Dependent Variables 



Variable 


Subscore 


1 Number of 


Number 


Title 


1 Items 


1 


General Course Attitude (GCA) 


8 


2 


Method of Instruction (MI) 


8 


3 


Course Content (CC) 


8 


4 


Interest -Attention (I-A) 


8 


S 


Instructor (I) 


8 


6 


Specific Items (SI) 


10 




Total 


50 



CEQ data for a total of 4,555 course sections taught during the period 
from the Fall of 1965-1966 to the Summer of 1970 at the University of Illinois 
and other institutions were placed on electromagnetic tape during the Spring 
semester of 1971. These data include the following information concerning 
each section: subscale means, total score mean, date (semester and year) of 
administration of the CEQ, number of students responding to the questionnaire, 
name of the institution (and campus, if applicable), course level, instructor*: 
rani;, sex of the instructor, and type of class. A listing in alphabetical 
order was made of the contents of this tape. This listing was scanned 
visually, and ail sections listed which met certain criteria were check- 
marked, indicating eligibility foj: inclusion in the study. Check-marked 
were all sections taught on the Urbana campus of the University of Illinois 
by instructors holding the regular ratk of teaching assistant through full 
professor at the end of atjj^ semester listed. A number of sections were 
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excluded solely because of missins course level and/or instructor rank* 
Ibst sections tausl^t before the Fall semester of 1966-*1967 were thus 
automatically excluded because the incidence of missinQ data uas highest 
during; the earlier semesters of CEQ administration* 

No instructor was represented more than once. That is, not more than 
one section taught by any one instructor uas included in the main study* 
For those instructors administering the C&Q to more than one section, the 
section to be included uas selected randomly from among all sections of a 
particular instructor's courses to uhich the CEQ uais given* Sections had 
been listed alphabetically by instructor's name to facilitate selecting 
sections of distinct professors* The 44& course sections finally included 
in the study^ therefore, provide data concerning courses taught by 448 
distinct professors* 

Desiytn 

The design of the study, then, is a 3 x 4 x S IIAMOVA viith six dependent 
variables* Analysis \;as conducted using the IBM 360 computer and the UMAVAC 
program originally developed by Finn (1966) and adapted for use at the 
University of Illinois by Dardrop and Bligh in 1969* The analysis was 
carried out directly on the means of the subscores, uith discriminant functions 
computed for those effects uhich v;ere significant. 

The specific hypotheses being tested are: 

(1) There are no significant differences in ratings assigned by 
students in small (l-'20) , medium (21*40) , and large (over 40) 
classes (class size) • 

(2) There are no significant differences in ratings assigned by 
students in 100- , 200- , 300- , and 400- and 500-level courses 
(course level) * 



(3) There are no significant differences in ratings received by 
teaching assistants , instructors, and assistant, associate, 
and full professors (instructor rank). 

The minimum level of significance necessary to reject each null hypothesis 
vas set at .05. 

Pilot St udy 

At the briginning of this investigation, a pilot study involving 205 
course sections t^as conducted* The purposes of the pilot study x;ere to pro*- 
vide the investigator \;ith insight into the nature of the concerns encounter*- 
ed v;hen actually converting data into a format required for pv^rformance of 
a MANOVA, particularly in connection uith the UMAVAC program, to enable her 
to successfully run such a program, to provide her x;ith first-hand experience 
in examining and interpreting the resultant output from the UMAVAC program, 
and to suggest refinements in general procedures of the main study. Results 
of the pilot study are not reported, since the main emphasis in this 
investigation is on the main study which is based on a greater number of 
course sections, uith sections selected for inclusion under slightly 
different conditions than during the pilot phase. 

RESULTS 

Table 2 presents the number of sections for each rank of instructor 
category by class size and course lev&l. This table shous that, in terms of 
class size, there v^as a total of 26 large, 156 medium, and 266 small course 
sections included. In terms of course level, there uas a total of A8 
400- and 500-level, 61 300-level, 102 200-level, and 237 ICO-level course 
sections. And in terms of instructor rank, there uas a total of 57 sections 
taught by full professors, and 62, 105, 60, and {54 sections taught by asso* 
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ci ate professors, assistant professors, instructors and teaching assistants, 
respectively* 



Table 2 

Number of Sections for Each Ranlc of Instructor 
Category by Class Size and Course Level 





Rank o£ 


Class 


Course Level 
















Instructor 


Size 


400 - 500 


300 


200 


100 


N 


Full 


Large 


G 


3 


1 


0 


4 


Professor 


Medium 


2 


8 


4 


1 


15 




Small 


11 


8 


13 


6 


38 


Associate 


Large 


0 


2 


0 


4 


6 


Professor 


Medium 


5 


5 


10 


2 


22 




Small 


10 


2 


12 


10 


34 


Assistant 


Large 


1 


3 


3 


1 


8 


Professor 


Medium 


12 


7 


8 


38 


65 




Small 


6 


13 


21 


72 


112 


Instructor 


Large 


0 


0 


0 


2 


2 




Medium 


0 


2 


2 


10 


14 




Small 


1 


3 


12 


28 


44 


Teaching 


Large 


0 


I 


1 


4 


6 


Assistant 


Medium 


0 


1 


6 


33 


40 




Small 


0 


3 


9 


26 


38 


N 


48 


61 


102 


237 


448 
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There were 11 empty cells, with 7 of those cells involving the highest 
(400- and 500-level) level courses. Such a finding is reasonable since only 
10.7 per cent (n - A8) of the sections represented are of the highest course 
level » 

As shown in Table 2, the highest cell frequency (n - 72) involves assist- 
ant professors teaching small lOO-level sections. There were 38 assistant 
professors and 33 teaching assistants teaching medium-sized 100-level sections* 
Only one large 400- and 500-level course was included. 

All in all, the entries in Table 2 seem reasonable and would probably 
be representative of frequencies in each cell if selection had been based 
on all the courses on the Urbana campus rather than those sections the 
instructor of which chose to administer the CEQ. 

The means for each cell on each of the dependent variables and the number 
of sections on which each mean is based are contained in Table 3. As indicated 
above, the highest possible sub-scale mean is 4*00, with higher scores indica- 
ting more positive student attitudes toward the course and its instructor. 
As noted at the bottom of this table, definition of each of the three factors 
and their respective levels is found on pages 5 and 6 of this report. 
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Table 3 
Means for Each Cell on Each 
of the Dependent Variables 



Cell* 


N** 


GCA 1 

1 


MI 


CC 


I-A 


I 


SI 


111 


26 


3.02 


2.79 


2-76 


2.65 


3.13 


2.82 


112 


28 


3.06 


2.39 


2.85 


2.81 


3.15 


2.91 


A. B«Cm 
113 


72 


3.02 


2-79 


2-79 


2.77 


3.16 


2.86 


114 


10 


2.93 


2-55 


2.66 


2. GO 


2.91 


2.77 


115 


6 


3.07 


2.59 


2.75 


2.67 


3.08 


2.34 


12 1 


5 


3.06 


2-75 


2.81 


2.74 


3.04 


2.92 


12 2 


12 


3.22 


2.99 


3.05 


2.99 


3.27 


3.04 


12 3 


21 


3.21 


2.93 


3.01 


2.92 


3.29 


2.97 


A-B*C . 
12 4 


12 


3.20 


2.93 


3.03 


2.92 


3-33 


3.01 


A.BX^ 
12 5 


13 


3.03 


2.69 


2.91 


2.68 


3.21 


2.88 


A.BX. 
13 1 


3 


3.40 


3.05 


3.04 


3.18 


3.39 


3.13 


13 2 


3 


3.31 


2.81 


2.93 


3.14 


3.37 


2.95 


13 3 


13 


3.27 


2.81 . 


2.99 


2.98 


3.26 


2.93 




2 


3.44 


3.14 


3.04 


3.19 


3.38 


3.01 


Ws 


8 


3.07 


2.62 


2-83 


2.64 


3-02 


2.33 


^1*4^ 


1 


3.05 


3.07 


3-02 


2.84 


3.20 


2.98 


W3 


6 


3.40 


3-18 


3.16 


3.20 


3.39 


3.17 




10 


3-30 


2.98 


2.9S 


3.08 


3.30 


2.94 




11 


3.33 


2.97 


3.04 


3.06 


3.23 


2.98 


Wl 


33 


3.00 


2.74 


2,81 


2.68 


3.10 


2.80 


W2 


10 


3.23 


3.08 


3.05 


3.06 


3.33 


2.99 


^2®lS 


38 


3.10 


2.82 


2.87 


2.75 


3.13 


2.91 
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(Table 3 Cont.) 



Cell* 


: N** 
A 


GCA 


m 


CC 


I -A 


I 


SI 






3.13 

1 

! 


2.50 


2.82 


1 

2.02 


3.16 


2.90 


^2»lS 


i 1 

1 
1 


j 3.G2 


2.75 


3.12 


3.01 


3.12 


3.14 




i ' 

1 


1 3.1G 

1 


2.99 


2.52 


2.00 


3.30 


2.55 




2 

1 


1 3.55 


3.43 


3.23 


3.36 


3.52 


3.23 




: 3 


I 3.20 


3.02 


3.07 


3.02 


3.35 


3.05 




i 

; 10 


2.95 


2.57 

1 


2.83 


2.57 


3.00 


2.84 


V2S 


i 

t 


2.03 


2.05 


2.70 


2.45 


3.19 


2.37 


Wl 


; 1 
1 


3.24 


2.57 


2.93 


3.00 


3.26 


2.51 




\ 2 


3.25 


2.41 


2.53 


2.70 


3.17 


3.C4 


A2B3C3 


1 ' 


3.05 


2.70 


2.01 


2.75 


3.09 


2.85 


^2®3^4 


! 3 

1 


3.25 


2.62 


2.38 


2.89 


3.16 


2.39 


A2B3C3 


1 

8 


3.32 


3.13 


2.56 


3.04 


3.45 


3.02 


W3 


12 

1 


3.22 


2.34 


2.53 


2.83 


3.23 


2.96 






3.06 


2.73 


2.83 


2.73 


3.26 


2.86 


^2^4^ 


1 

2 1 

i 


3.12 


2.65 


2.86 


2.69 


3.13 


2.73 




i 

4 


3.03 

] 


2.84 


2.86 


2.33 


3.19 


2.03 


^3«lS 


2 


3.27 


2.92 


2.90 


3.00 


3.05 


2.54 




1 


3.13 


2.95 


2.77 


2.82 


3,15 


2.77 






3.05 


2.63 


2.82 


2.68 


3.15 


2.76 


Wl 


1 


2.97 


2.87 


2.80 


2.73 


3.01 


2.82 


A3B2C3 


3 


3.28 


2.97 


3.13 


2.95 


3.16 


3.03 


A3B2C5 


1 


3.35 


2.92 


3.01 


2.93 


3.12 


3.95 


A3B3C, 


1 


2.85 


2.29 


2.80 


2.41 


2.91 


2.50 
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(Table 3 cont.) 



Cell* N** GCA 



A3B3C3 

A3B3C5 

I 



MI 



3 I 2.88 I 
2 j 3.25 
3.06 
1 2.74 



2.20 
2.88 
2.62 



CC 



I -A 



4- 



3.02 
2.67 



2.74 3.05 



2.17 ! 2.49 2.22 i 2.78 



SI } 



< ■: 
2.67 ■ 2.39 i 2.69 . 2.74 



2.92 1 3.26 ! 2.92 



2.83 
2.71 



*See pages 5-6 for definition of Factors A, B, and C and their respective 
levels . 

**N « nunber of sections in each cell. 



Tables 4, 5, and 6 present results of tests of significance for each 
of the three main effects, and Tables 8, 9, and 10 present results for 
each of the t\«>-way interactions. The only main effect found to be signi- 
ficant was the level effect. In addition, the size x level and the size 
X rank interactions were significant. 

In Table 4, the size effect is shown to be nonsignificant (Multivariate F 
1.48, d.f. « 12 and 816, p <.13). Thus, there were no significant differences 
in ratings in terms of class size. In this and other tables reporting results 
of tests of significance, results of Multivariate F, Univariate F, and Step- 
down F tests are reported. In Table 4, it is evident that none of the Uni- 
variate F results are significant, but that the value of 3.40 indiceted for 
the Step-down F for the Interest-Attention variable is significant at the .05 
level. Results of the univf.viate tests of significance show the effect of the 
individual dependent variables. Thus, for the size effect, none of the six 
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dependent variables taken singly produced significant differences in CEQ 
ratings between small, medium, and large course sections. Step-dov;n F 
results indicate effects of each dependent variable when variance attributed 
to other dependent variables is partialed out* Significant values of Step- 
doxm F indicate the variable(s) primarily responsible for significant values 
of the (overall) Multivariate F. 



Table 4 
Tests of Significance 
for the Size Effect 





Dependent Variable 


Univariate F 
(dfj^=2, df^-413) 


Step-dovm F 
(dfj^«2, df^=413) 


Multivariate F 
(dfj^»12, df^=816) 


General Course Attitude 


.10 


.10 




Method of Instruction 


1.20 


2.76 




Course Content 


.AO 


1.C5 


F «= 1.48 


Interest-Attention 


.82 


3.40* 


p< .13 


Instructor 


1.89 


.61 


Specific Items 


.12 


.94 


[ ■ .. .... .. ■ , . 



*p < .05 



Table 5 sho\7s that the level effect x;as highly significant (Multivariate 
F » 6.A0, d.f. = 18 and 1154.4834, p < .0001) . The variables primarily responsible 
for the signiticant level effect arc General Course Attitude (Step-doxm F = 6.31 > 
p< .001)^ Method of Instruction (Step-doun F = 11.08, p< .001), and Course 
Content (Step-doxra F « 13.30, p< .001. The step-doxm F values for the Interest- 
Attention and Instructor variables \;ere also significant at the .05 level, t;ith 
the only variable for v;hom a nonsignificant value of Step-down F was found 
being Specific Items* There were two significant discriminant functions 
(Bartlett^s X^^ « 110.6064, d.f. « 18, p < .00001; Bartlett's X^^ ^0*<>8» d-f- 
10, p< .00001) for the level effect: 
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-•7ax^ ^ 2,20X2 * ^^^3 " -^^4 • -^^S ^ -^^6 

or 

.78 (General Course Attitude) +2,20 (Itethod of Instruction) - 
•83 (Course Content) - .24 (Interest-Attention) - .60 (Instructor) - 
• 28 (Specific Items). 

Y2 « 1-llX^ - 1.01X2 - 1.67X3 1.40X^ ^ .24X^ - .2/iXg. 

or 

Y2 « 1.11 (General Course Attitude) - 1,01 (Method of Instruction) - 

1-67 (Course Content) +1.40 (Interest-Attention) + .24 (Instructor) 
.24 (Specific Items) • 

A plot of these functions is presented in Figure 1. The first function, 
Yj^, is based mostly on high scores on Method of Instruction and low scores on 
Course Content and General Course Attitude* On this function, 100- and 400- 
and 500-level course sections \;ere very similar. The 300-level sections were 
much lower on this function than courses of other levels, with the 200-level 
sections intermediate (between 100- and 400-level and 300-level sections) . 

The second function, Y^, is based primarily on high scores on General 
Course Attitude and Interest-Attention and low scores on Course Content. Iliis 
function separates 200-level sections from all others » and tends to spread out 
the remaining three levels from one another, but not very far, with 300-level 
highest, 100-level next, then 400- and 500-level, and finally 200-level 
sections lowest. 

Thus, for the first function for the level effect, high Method of Instruc- 
tion and low Course Content and General Course Attitude ratings separate 100- 
and 400- and 500-level courses from courses of the other two levels, with 300- 
level courses lowest and 200-level courses intermediate on this function. 
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I 



■1.0 



300 -level 
(-1.0, .61) 



100- level 
(-.02, .36) 



T" 

.5 



200- level 
(-.39, -.69) 



II 

r 1.0 



- .5 



400- and 500- level 
(0,0) 

1 



- -.5 



L-1.0 



-1 ^ 

1.0 



Figure 1. Centroids of the four groups based on course level in 
the discriminant space. 
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And for the second function, hish General Course Attitude and Interest-Attention 
and lou Course Content ratings separate 200- level sections from all the rest, 
xiith 300-level sections highest on this function. 

Table 6 shous results of the tests of significance for the ranlc of instructor 
effect. There uere no significant differences by rank Qlultivariate F - 1*35 , 
d.f. « 24 and 1A24.5520, p< .13). Ho-iever, as the univariate tests of signifi- 
cance shov7ed, if c:hc variables X7ere examined one at a time, significant differences 
would have been found anK>ng Method of Instruction, Course Content, Interest- 
Attention, and Specific Items x^ith Univariate F values of 2*^^, 2.71, 2*75, and 
3.07, respectively, all significant at the .05 level. Table 7, hox;ever, shoxis 
that there are high intcrcorrelations among the six dependent variables (r - .67 
- .91). Thus, the four significant differences found during univariate tests 
are artifacts of the high intcrcorrelations among the CEQ subscales. When 
the inter correlations of each of the four significant (during univariate analysis) 
scales arc taken into account, in addition to their intcrcorrelations x;ith the 
two remaining scales, the remaining differences are no longer found to be 
significant. Thus, although univariate test.^ had revealed significant rank 
differences, multivariate tests did not. The multivariate tests take into 
account the relationships among the dependent variables, thus presenting a 
more accurate account of events being studied. 
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Table 6 
Tests of Significance 
for the Rank Effect 



Dependent Variable 


Univariate F 
(dfj^=4, df^=4i3) 


Step -down F 
(df^=A, df^=Al3) 


Multivariate F 
(df,=24, df ^=1424. 5520) 


General Course Attitude 


1.23 


1.23 




Method of Instruction 


2.48* 


2.17 


F = 1.35 


Course Content 


2.71* 


1.27 


p < .13 


Interest-Attention 


2.75* 


1.65 




Instructor 


1.00 


.76 




Specific Items 


3,07* 


1 1.02 





Table 7 

Intercorrelations Among the Six Dependent Variables 



Dependent Variable 


1 


2 


3 


4 


5 


6 


1 


General Course Attitude 


1.00 












2 


Method of Instruction 


.85 


1.00 










3 


Course Content 


.86 


.84 


1.00 








4 


Interest-Attention 


.91 


.89 


.87 


1.00 






5 


Instructor 


.67 


.81 


.72 


.73 


1.00 




6 


Specific Items 


.77 


.80 


.86 


.78 


.69 


1.00 



* p < •OS 
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Tests o£ significance £or the size x level interaction are shown in 

Table 8. The results vere significant at the .03 level (Multivariate F « 

1A9^ d.t. "* 36 and 1794.4136) • The variables primarily responsible for 

the significant interaction between size and level are Course Content and 

Specific Items • The one significant discriminant function for this inter- 

2 

action effect (Bartlett*sX « 53.53, d.f. « 36, p < .04) is: 

« .37X, + .26Xo - 2.08X, - .28X, + .03K. + 1.27X. 
1 1 2 3 4 5 6 

or 

« .37 (General Course Attitude) + .26 (Method of Instruction) - 2.08 
(Course Content) - .28 (Interest-Attention) + .03 (Instructor) 
1.27 (Specific Items). 
This function, then, is based primarily on low scores on Course Content and 
high scores on Specific Items. The distribution of sections according to 
class size and course level is found in Figure 2. 



large and small 200.1evel /"'^'^?;°:Jr?Li 

500-Xevel \ | , , 

medium 100-level 



^ / smal 

\\\ 



s 

II M I 
II 




II I I I 
ii I i 



I 1 r « 1 1 ' I — ^ — T~ 

.25 .50 .75 1.0 1.25 1.5 1.75 2.0 2.25 2.5 



medium 300- 

level 
small 100- 
level 



Figure 2. Canonical weights of sections according to class size and 
course level. 
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This function, Y^, seems to primarily separate large ACO- and 500*levcl 
sections from the rest of the group. Large AOC* and SOO-Ievel courses are 
quite separate from the rest of the sections and lowest on this function. In 
addition, it seems to separate 200-level sections from 100* and 300-level 
sections. The medium^slzed sections cannot be completely separated from the 
small sections because the medium classes are embedded x^ithin the small group. 

Group centroids on Yj^ are given belox; in Table 9. The numbers in parentheses 
indicate the number of sections on which each centroid is based. It should be 
noted that there x*as only one large AOO- and 500-level section, a fact which 
makes Interpretation of this function rather difficult. The difference 
between the centroids of small and medium sections for each level is given 
so that interaction effects can be viex^ed more clearly. 



Table 9 
Group Centroids 



[ Course 


Class Size 


Level 


Small 


Medium 


Small - Hediuffl 


100 


^•^^(142) 




-l-.Al 


200 






-.10 


300 


^•52(29) 


2-^^23) 


-.26 



These results shoxi that at the 100-level, instructors of small classes 
received higher ratings on the CEQ on this function. At the 200-level, there 
was not much difference between ratings received by instructors of small and 
medium classes. At the 300-Xevel, however, instructors of mediua-sized 
classes received higher ratings than did instructors of small classes. 
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Figure 3 sho^9s a plot of the centroids for saall and mediun classes at 
each of the three course levels. In this figure it can be seen that the lines 
for small and medium classes cross over indicating significant interaction 
between class size and course level. 

Thus, the significant interaction between class size and course level 
shovs that small classes at the 100* level are rated higher on Specific Iteos 
and louer on Course Content than are loedium classes at the same level* Medium 
classes at the 300-level are rated higher on Specific Items and louer on 
Course Con^tent than are small classes at that level* There seems to be no 
appreciable difference betxieen ratings assigned by small medium classes at 
the 200-level. 

Table 10 shoiis that the size x rani: interaction t/as significant (tftilti- 
variate F = 1.47, d.f. = 4G and 2011»5935, p <.02). The variables priaaarily 
responsible for the significant interaction are Method of Instruction (Step- 
do\m F = 3.45, p < .05) and Specific Items (Stcp-doun F =^ 3.45, p < .001) • It 
should be noted that although the values of Step-^dotm F for these tvio variables 
t;ere significant, significant values for Univariate F had not been found for 

any of the six dependent variables. The one significant discriminant function 

2 

for this interaction effect (Bartlett's X = 70.35, d.f. = 48, p < .02) is: 

= -•OaX, -S- .2(5K^ - 2.13X^ + •55X, - .0«^ > 1.67X^ 
I 1 2 3 4 3 0 

or 

Yj^ = -.03 (General Course Attitude) .28 (Method of Instruction) - 2,13 
(Course Content) .55 (Interest-Attention) - .04 (Instructor) ^ 
1.67 (Specific Items). 
This function, then is based primarily on lov; scores on Course Content and 
high scores on Specific Items. The distribution of sections according to 
class size and instructor rank on this function is shovm in Figure 4, viith 
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Small Classes 
Hedium classes 




Figure 3. Group centroids on the significant discriminant function 
for the interaction between class size and course level « 
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size indicated by L, and S, for large, medium, and small, respectively, 
and rank, indicated by 1, 2, 3, 4, and 5, for teaching assistant, instructor, 
assistant professor, associate professor, and full professor, respectively: 




Figure 4. Canonical weights of instructor rank and section size. 

Ihifl function seems to clearly separate large sections taught by full 
professors from all the remaining sections. In addition, medium sections 
are separated from small sections. Large sections taught by full professors 
were lowest, medium-sized sections were next (intermediate), and small 
sections ware highest, on this function. Of those teaching meditmi-sized 
sections, instructors and teaching assistants were similar on this function, 
as were assistant and associate professors. Of those teaching small sections 
assistant professors and teaching assistants were quite similar on this 
function. 

Group centroids on are given in Table 11, again with numbers in 
parentheses indicating the number of. sections on which each centroid is 
based. There were four full professors teaching large sections. 
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Differences between centroids of small and mediiim*- sized groups for 
each instructor rank are also included. 

Table 11 
Group Centroids 





Instructor 




Class 


Size 




Rank 


Small 


Medium 


Small - 


Medium 


Teaching Assistant 






+1 


.19 


Instructor 




2-7^14) 


+1 


.07 


Assistant Professor 


3 93 

(112) 




+ 


.89 


Associate Professor 


^•'^^34) 


2- 2^^(22) 


+ 


.94 



These results .show that for each rank of instructor level, instructors 
of small classes received higher ratings on function Yy That is, for each 
rank of instructor, lower ratings on Course Content and higher ratings on 
Specific Items were assigned by small than by medium classes. In addition, 
generally it seems that the lower the rank of instructor, the more favorable 
were ratings assigned by small as compared to medium sections. 

A plot of the centroids for small and medium sections at each of the 
four rank of instructor levels is presented in Figure 5. Tliis figure shows 
that for each rank level, instructors of small classes received higher 
ratings on this particular function. 

Table 12 indicates that the level x rank interaction was not significant 
(Multivariate F « 1.31, d.£. = 66 and 2188.6018, p <.06). There were no 
significant values of Univariate F for this interaction, but values of Step- 
down P for two of the variables (Method of Instruction-^-Step-down F = 2.22 
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Figure 5 . Group centroids on the significant discriminant function 
for the interaction between class size and instructor rank (main study) . 
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Specific Items— Step-down F = 1.89) were significant at the .05 level. 
Therefore, if the value of Multivariate F had been significant; such 
significance would have been attributed primarily to Method of Instruction 
and Specific Items, the two variables which probably would have received 
the most weight in the discriminant function (s) found to be significant 
for this effect. 

Table 12 
Tests of Significance for 
the Level X Rank Interaction 



Dependent Variable 


Univariate F 


Step-down F 


Multivariate F 




(dfj^«ll, df^-413) 


(dfj^oll, df^-413) 


(dfj^=66, df ^-2188. 6018) 


General Course Attitude 


.95 


.95 




Method of Instruction 


1.00 


2.22* 


F « 1.31 


Course Content 


.80 


.88 


p <.06 


Interest -Attention 


.81 


.58 




Instructor 


.78 


1.33 




Specific Items 


.74 


1.89* 





» p <,05 
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SUMMARY AND CONCLUSIONS 

The null hypotheses for the size and rank main effects tiere supported 
in the present ^j-udy. There were, as hypothesized, no sicnificant differences 
in ratings assicned by students in small (1-20), medium (21-40), and large 
(over 40) classes, (size alone), or received by teaching assistants, instructors, 
and assistant, associate, and full professors (rank alone). The hypothesis 
for the level effect, however, was rejected, since there were highly significant 
differences in ratings assigned by students in 100-, 200- , 300-, and 400- and 
500-level courses. Two significant discriminant functions were computed for 
the level effect, pointing out the extent and direction of differences among 
the groups in terms of course level. The first function was based primarily 
on high scores on the Method of Instruction and lo\/ scores on the Course 
Content and General Course Attitude subscales. This function primarily 
separated 100- and 400- and 500-lcvel courses which were highest, 200-level 
courses which were intermediate, and 300-level courses which "ere lowest on 
the function. Thus, for the first function for the level effect, 100- and 
400- and 500-level courses were rated highest on the Method of Instruction 
and loxiest on the Course Content and General Course Attitude subscales, and 
300-level courses were rated lowest on the Method of Instruction and highest 
on the Course Content and General Course Attitude subscales. The second 
significant discriminant function for the level effect was based primarily 
on high scores on General Course Attitude and Interest-Attention and low 
scores on Course Content. This function separated 200-level sections from 
all the rest of the sections, with 300-level courses highest on the function. 
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Of the interaction cffccto, the size x level and the size x rani: effects 
were significant, whereas the level x rank interaction was not. The one 
significant discrininant function for the size x level interaction primarily 
separated large AOO- and 500-level sections (loviest on this function) , as 
t;ell as 200-level sections (intermediate on this function) from the rest of 
the groups, uith nedium-sised groups embedded within those of snail size. 
It was found that at the 100-level snail classes were rated higher on the 
Specific Items and lower on the Course Content subscales than were nediun- 
sized classes; at the 200-level, there was no appreciable difference between 
snail and nediura-sized sections; and at the 300-level, laediun-sised classes 
were rated higher on the Specific Iteno and lower on the Course Content 
subscale than were small classes. 

The one significant discriminant function for the size x ranh interaction 
was based primarily on low Course Content and high Specific Items subscale 
scores. For each rank, snail classes were rated higher than medium-sized 
classes on this function, and the lower the instructor's rank (with the 
exception of assistant and associate professors which were found. to be 
similar) the more favorable were ratings assigned by snail as compared to 
medium-sized sections. 

The fact that the multivariate F test for the ranlc effect did not 
reveal statistical significance is of great practical significance in this 
investigation. Univariate tests conducted in this and in other (bivariate) 
studies involving the CEQ have slio.m significant differences in terms of 
ranlc, with higher ranking instructors tending to receive higher ratings 
on the CEQ, but multivariate tests did not. This discrepancy wao attributed 
to the high inter-correlations among the CEQ subscales. The multivariate 
tests used herein took into account the relationships among the dependent 
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variables, thus presenting a niorc accurate account of the relationship 
between instructor rank and CEQ ratings. 

It should be noted that the results reported in this investigation 
arc based upon that group of distinct instructors at the Urbana campus of the 
University of Illinois foi* \jhom complete and relevant CEQ data were available* 
Because of the voluntary nature of administration of the CEQ, generali::ations 
concerning teaching effectiveness cannot be made concerning the campus as a 
whole. Also, since only the Urbana campus was included in the study, 
generalisations cannot be made concerning relationships betxjcen CEQ ratings 
and instructor's rank, class size, and level of course at other institutions 
or campuses of this institution. In addition, results obtained must be 
regarded with caution because of the large number of cases involved. That 
is, significant results obtained herein may possibly be due largely to the 
sheer magnitude of the degrees of freedom involved. Another possible 
limitation of this study concerns the classification of sections into class 
si*zc« The classification of class size into the three particular categories 
used was based on logical analysis of the data of the pilot study. This 
classification, like any, causes loss of data which may have affected the 
results of the study. The points of trichotomization selected could have 
greatly affected the conclusions reached. Another investigator might have 
chosen a different means of classifying sections according to size. Thus, 
a suggestion for further study is investigation of the effect of different 
classification schemes for class size, including classification into other 
than three levels. 

Further investigation into reasons for the actual results obtained 
should also be made. As indicated, univariate analyses have tended to show 
differences in CEQ ratings in terns of instructor rani;. However, as pointed 
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out by persons such as Cattell (1966), French (1970), Fruchter (1969), 
Jones (1906), Roccboon (1966), and Tatsuoka (1969), human behavior is 
highly conplcx, and therefore, the multivariate approach is preferable 
to conducting a series of univariate analyses of bivariate data in order 
to determine relationships in certain areas of interest. This study has 
been conducted as an attempt to shou that the multivariate approach is 
indeed, as pointed out by Cattell (1966), more economical, more consistent, 
and more comprehensive than is the utilization of data from a series of 
bivariate studies in leading to the formulation of more adequate and 
representative conceptions of the state of events in a partic^jlar area* 
In this study, an attempt has been made to shov; that simultaneous 
nmltivariate design does tend to eliminate the problem of statistical 
dependence among variables v;hich "upsets the significance levels" uhen 
using a "scries of univariate tests" (Tatsuoka, 1969) . As pointed out 
by Tatsuoka (1969) and others, the multivariate approach is extremely 
powerful uhen applied correctly. MANOVA in particular, is useful in 
detecting the existence of significance differences among groups on 
many variables simultaneously, and in yielding information through the 
use of discriminant functions computed for effects found to be significant 
concerning the extent and direction of those significant differences • 
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